Progress of Theoretical Physics Supplement 



1 



Finite-Temperature Mott Transition in Two-Dimensional 
Frustrated Hubbard Models 

Takuma Ohashi 1>2 ), Tsutomu Momoi 2 \ 
Hirokazu Tsunetsugu 3 ) and Norio Kawakami 4 ) 

1 Department of Physics, Osaka University, Toyonaka, Osaka 560-0043, Japan. 
2 Condensed Matter Theory Laboratory, RIKEN, Wako, Saitama 351-0198, Japan. 
^Institute for Solid State Physics, University of Tokyo, Kashiwa, Chiba 277-8581, 

Japan. 

^ Department of Physics, Kyoto University, Kyoto 606-8502, Japan 

We investigate the Hubbard model on two typical frustrated lattices in two dimensions, 
the kagome lattice and the anisotropic triangular lattice, by means of the cellular dynam- 
ical mean field theory. We show that the metallic phase is stabilized up to fairly large 
Hubbard interactions under strong geometrical frustration in both cases, which results in 
heavy fermion behavior and several anomalous properties around the Mott transition point. 
In particular, for the anisotropic triangular lattice, we find novel reentrant behavior in the 
Mott transition in the moderately frustrated parameter regime, which is caused by the com- 
petition between Fermi-liquid formation and magnetic correlations. It is demonstrated that 
the reentrant behavior is a generic feature inherent in the Mott transition with intermedi- 
ate geometrical frustration, and indeed in accordance with recent experimental findings for 
organic materials. 

§1. Introduction 

Geometrical frustration has attracted much interest in strongly correlated elec- 
tron systems. Among a number of intriguing phenomena, the observation of heavy 
fermion behavior in L1V2O4 1 )' 2 ) with the pyrochlore lattice structure has activated 
theoretical studies of electron correlations with geometrical frustration. Also, the 
discovery of superconductivity in the triangular-lattice oxide Na x Co02 • t/E^O 3 ) and 
the /3-pyrochlore osmate AO^Oq {A =K, 4 ) Rb, 5 ^ Cs 6 ^) has stimulated further inves- 
tigations of frustrated electron systems. These intensive studies have revealed new 
aspects of the Mott transition for geometrically frustrated electrons. In particular, 
an experimental demonstration of the spin liquid ground state in the Mott insulat- 
ing phase in the organic material k-(BEDT-TTF)2Cu2CN3 7 ) with triangular lattice 
structure, raises a theoretical challenge in the physics of geometrically frustrated 
electron systems. Other materials found recently, such as the pyrochlore Kondo lat- 
tice compound Pr2ir2 07 8 ) and the hyperkagome compound Na^^Os, 9 ' also provide 
new examples of frustrated electron systems with nonmagnetic ground state. In 
this paper, we will investigate the two fundamental frustrated systems, the kagome 
lattice and the anisotropic triangular lattice, which we will briefly explain below. 

1.1. Kagome lattice 

The kagome lattice is one of typical frustrated systems and it shares some 
essential properties of the pyrochlore lattice. Antiferromagnetic spin systems on 
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this lattice have been intensively studied and many unusual properties have been 
found. 10 )' 11 ) Theoretical studies of the 5 = 1/2 Heisenberg antiferromagnet has sug- 
gested the realization of a nonmagnetic ground state and the existence of anomalous 
singlet excitations within the singlet-triplet gap due to strong frustration. Therefore, 
it is natural to ask what kind of quasiparticle dynamics these unusual properties in- 
duce if electrons become itinerant. An interesting example of the itinerant kagome 
systems may be a superconducting compound Na^OoC^ • 2/H2O. It has been sug- 
gested that an effective model of this material can be regarded as a correlated electron 
system on the kagome lattice by properly considering anisotropic hopping matrix el- 
ements of the cobalt 3d orbitals. 12 ) Also, the hyperkagome compound NaztlrsOs is a 
three dimensional analog of the kagome lattice electron system. The issue of electron 
correlations for the kagome lattice was addressed recently in the studies by using the 
fluctuation-exchange (FLEX) approximation 13 -* and quantum Monte Carlo (QMC) 
method. 14 ^ These studies focused on electron correlations in the metallic regime, 
and the nature of the Mott transition has not been clarified. We shall investigate in 
this paper the kagome lattice electron system with particular emphasis on the Mott 
transition under the influence of strong frustration. 

1.2. Triangular lattice 

Another remarkable example is a triangular electron system with strong cor- 
relations, in which the metal-insulator transition has intensively studied. This has 
particularly been stimulated by various interesting phenomena recently found in 
organic materials k-(BEDT-TTF)2A around the Mott transition, such as a spin liq- 
uid state, unconventional superconductivity, etc. 7 )' 15 )' 16 ) A possible nonmagnetic 
ground state is found in the triangular lattice Hubbard model by the path integral 
renormalization group study, which provides a powerful numerical treatment of the 
frustrated electron systems, 17 ) and the correlated electrons on the anisotropic trian- 
gular lattice have been intensively studied so far. 18 ) -27 ) The effects of geometrical 
frustration on finite-temperature (T) Mott transition, however, have not yet been 
sufficiently understood. One of the interesting and nontrivial features of the finite- T 
Mott transition is a reentrant behavior observed in the frustrated organic material 
k-(BEDT-TTF) 2 Cu[N(CN) 2 ]C1 under pressure. 15 )' 16 ) With lowering temperature, 
it once undergoes a transition from Mott insulator to metal, and then reenters the 
paramagnetic insulating phase at a much lower temperature. This reentrant behav- 
ior is quite different from the nonreentrant behavior of Mott transition in the three 
dimensional systems, such as V2O3, and is expected to be a new aspect of the geo- 
metrical frustration and possibly low-dimensionality. We will address this problem 
in this paper. 

1.3. Theoretical approach 

In order to investigate the Mott transition in the kagome and triangular electron 
systems, we need advanced theoretical methods. Among many approaches for cor- 
related electron systems, the dynamical mean field theory (DMFT) 28 )~ 30 ) has given 
substantial theoretical progress in understanding the Mott transition, 31 ) and it has 
also clarified various interesting phenomena 29 )' 32 ) in the strongly correlated electron 
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systems, such as magnetism, 33 )~ 39 ) heavy fermion formation, 40 )~ 45 ) orbital physics in 
the multiband systems, 46 )~ 54 ) etc. Recently, DMFT has been also applied to cold 
atoms in an optical lattice, 55 )~ 58 ) some inhomogeneous systems, 59 )~ 61 ) and photo- 
excited semiconductors. 62 )' 63 ^ However, DMFT does not take account of spatially 
extended correlations, which should be included for the systems under consideration 
in this paper. Therefore, it is desirable to study the Mott transition by employing 
another appropriate method which properly incorporate spatially extended corre- 
lations and geometrical frustration. Cluster extensions of DMFT 64 )~ 68 ) or the self 
energy functional approach 70 ) are candidates for this purpose. Recently developed 
diagrammatic extensions of DMFT 71 )' 72 ) might also treat geometrical frustration 
by incorporating the k-dependence of the self-energy. Here we will use a cluster 
extension of DMFT, the cellular dynamical mean field theory (CDMFT). 65 )' 66 ) 

1.4. Purpose of the paper 

In this paper, we give a brief review of our recent studies on the Mott transitions 
in the Hubbard model on the geometrically frustrated kagome 73 ) -75 ) and anisotropic 
triangular lattices 76 ) by means of CDMFT combined with QMC. 77 ) We investigate 
these models separately to discuss properties characteristic of each system, and then 
deduce common properties inherent in frustrated electron systems. In both models, 
we find that the metallic phase is stable up to fairly large Hubbard interactions 
under strong geometrical frustration, giving rise to the heavy fermion behavior near 
the Mott transition. In the kagome lattice, several anomalous properties of spin 
correlation functions, such as nonmonotonic temperature dependence, emerge around 
the Mott transition. For the anisotropic triangular lattice, we discover more striking 
behavior in the Mott transition. Namely, in moderately frustrated cases, the finite- 
T Mott transition shows a reentrant behavior, which is consistent with experiments 
in some organic materials. 15 )' 16 ) We demonstrate that the reentrant behavior is a 
characteristic feature inherent in the Mott transition with geometrical frustration, 
and thus can be experimentally observed in various frustrated electron systems. 

The paper is organized as follows. In the next section, we introduce the model 
Hamiltonian and briefly explain the framework of CDMFT. We first study the Mott 
transition in the kagome lattice Hubbard model and elucidate some anomalous prop- 
erties appearing near the transition point in §3) In 21 we then investigate the reen- 
trant Mott transition on the anisotropic triangular lattice. A brief summary is given 
in the last section. 

§2. Model and method 

We consider the standard Hubbard model on the kagome lattice (see Fig. [2{a)) 
and the anisotropic triangular lattice (see Fig. [T3j) . 

H = Y ' './'-'> + u Yl ni T n *i ' i 2 ' 1 ) 

i,j,cr i 

with ni a = c\ a Ci a , where c\ a (cj CT ) creates (annihilates) an electron with spin a at site 
i. Here, i,j = 1, 2, • • • , N, and N is the total number of sites. The hopping matrix 
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Fig. 1. (Color online) Sketch of the original lattice (a), superlattice (b) and effective cluster model 
(c) for the cluster size N c = 4. 



element and the Hubbard interaction are denoted as Uj and U, respectively. The 
explicit form of will be given for each model below. Note that in both models, 
a triangular structure of the unit cell, which is a source of strong frustration, plays 
a crucial role in controlling the nature of Mott transition. Therefore, theoretical 
methods beyond DMFT are necessary to incorporate spatially extended electron 
correlations. To this end, we here use CDMFT, which has been successfully applied 
to frustrated systems such as the Hubbard model on the triangular lattice 21 )' 24 )' 25 ) 
and the kagome lattice. 73 ** 

2.1. Cellular dynamical mean field theory 

In CDMFT, the original lattice is regarded as a superlattice consisting of small 
clusters, as shown in Fig. [1] (b). Here, we define the cluster size as iV c . The model 
(|2-ip is rewritten as, 



l,m,-y,S,a 2,7 



(2-2) 



where I and m are cluster indices, I, m = 1, 2, • • ■ , N/N c , and 7 and 5 are sublattice 
indices, 7, 5 = 1, 2, • • • ,N C . Using a standard DMFT procedure with the inter-cluster 
hopping scaled as t~ 5 (Z, m) — > t~ t s(l, m)/y/d, the original model is then mapped onto 
an effective cluster model consisting of a cluster coupled to the self-consistently 
determined medium, as illustrated in Fig. [T](c). The corresponding action reads 



S cS = f dr f dr> £ ct CT (r) (CT 1 ) (r - r') c &a (r') 

+ U dr V n 7T (r) n 7i (r) . 
J 



(2-3) 



Here, (3 = 1/T. Given the Green's function for the effective medium, Q a , we can 
compute the cluster Green's function G a by solving the effective cluster model with 
QMC method, 77 ^ and then we obtain the cluster self-energy E a . Here, Q a , G a , 
and E a are A^ c x N c matrices. In order to reduce errors due to finite time slices in 
QMC, we exploit an interpolation scheme based on a high-frequency expansion of 
the discrete imaginary-time Green's function. 7S ) The effective medium Q a is then 
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updated by the Dyson equation, 
g- 1 {iuj n ) = 



ILOr 



(2-4) 



(2-5) 



where [i is the chemical potential and t(k) is the Fourier-transformed hopping matrix 
for the superlattice, 



g a ( k : iuj n ) = ito n + /j, - i ( k) - S a (iu n ) 



t 7 S I k 



N 



c ^ e -k-( ri -r m ) 

Lm 



tjs (l,m) . 



(2-6) 



Here the summation of k is taken over the reduced Brillouin zone of the superlattice. 

2.2. Wave-vector dependent properties 

Within CDMFT, the single-electron Green's function for wave vector k is given 



as. 



G k (iuj n ) 



1 



e ik-(r 7 -r<5) 

7,<5 



iuj n + fj, — t (k) — E (ico r , 



-i 



7<5 



(2-7) 



where k is the wave vector in the original Brillouin zone and r 7 , label clus- 
ter sites. 69 ) We calculate the imaginary time Green's function (r) and obtain 
the spectrum (w) = — ImGk (uj + iO) /ir using the maximum entropy method 
(MEM). 80 ) 

It is also possible to compute the wave- vector dependence of various two-electron 
Green's functions with including vertex corrections. Here, we investigate the q- 
dependent static spin susceptibility defined as, 

X 7 *(q) = f f dr £ ( C t ^ (t) W (r) (0) (0)) . (2-8) 

k,k' 

In order to incorporate the vertex correction into the susceptibility, we consider the 
two-electron Green's function in the effective cluster model ()2-3[) . 



1 

PJo 



P rP rP rP 



JO Jo 



dTidT2dT^dT4 



x e -M(r l -^) e -i Um (v i -rA) CfrS (ri) T2j T3j T4 ) ( 

C 7 5 (ri, r 2 , r 3 , r 4 ) = (|t t c 7T (n) c li (r 2 ) (r 3 ) c<5 T (r 4 )^> . 



(2-9) 
(2-10) 



We first calculate the cluster two-electron Green's function (|2-10p by QMC and 
extract the vertex function r^s (iu)i,iuj m ) via the Bethe-Salpeter equation, 



r = c° -C 



-i 



(2-11) 
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Fig. 2. (Color online) (a) Sketch of the kagome 
lattice and (b) the first Brillouin zone. 
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Fig. 3. (Color online) Phase diagram of the 
kagome lattice Hubbard model. The dashed 
lines are guide to eye. 



where C° is the bare cluster two-electron Green's function, 



Nc 
N 



^2^si (k 



lUl 



Nc 
N 



9W ( k : 



Si, 



(2-12) 



Here, 0°, C and P are N c Nf x N c Nf matrices, and Nf is the number of the Mat- 
subara frequency. On the other hand, the bare two-electron Green's function in the 
lattice system is calculated as, 



C° 5 (q : iui,iu ri 



' N 



y^g 7 ^| (k + q : iujA g^s] (k : iuA 5^ m . (2-13) 



By using Eqs. (|2-11|) and (|2-13|) . we can compute the lattice two-electron Green's 
function, 



C(q) 



(2-14) 



C0(q) -r . 

Taking account of the phase factor, we finally obtain the q-dependent susceptibility, 
X7*(q) = ^2 E C ^ : ^^m) e~^- r *\ (2-15) 



Lm 



§3. Kagome lattice system 

In this section, we investigate the Mott transition in the kagome lattice Hubbard 
model by means of CDMFT and determine the phase diagram for the Mott transition. 
The result is shown in Fig. [3j 

Let us consider the Hubbard model (|2-1|) with nearest-neighbor hopping on the 
kagome lattice (see Fig. 0(a)), 



U 



—t (t > 0) (site i an j : nearest neighbors) 
(otherwise) 



(3-1) 
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Fig. 4. (Color online) Double occupancy as a 
function of interaction strength U/W. At 
T/W = 0.0125, we can see the discontinu- 
ity with hysteresis, indicating the first-order 
Mott transition. 
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Fig. 5. (Color online) Derivatives of double 
occupancy dD occ . jdU . We define the 
crossover point U* in the phase diagram 
Fig. [3] by U that gives a maximum of 
dDocc. /dU. 



The band width is W = 6t and we will use it as an energy unit. Unit cell of the 
kagome lattice has three sites and they are labeled by 1, 2, and 3, as shown in Fig. 
[2(a). We choose this unit cell as a cluster for the CDMFT approach and map the 
system to an effective cluster model. Self-consistent solution of self-energy matrix 
is obtained by means of iterative procedure explained in the previous section, and 
twenty-times iterations are sufficient to achieve satisfactory convergence. In each 
iteration, the local single- and two-electron Green's function for the effective model 
are calculated by QMC, where we typically use 10 6 QMC sweeps and Trotter time 
slices L = 2W (3 to reach sufficient computational accuracy. 

3.1. Mott transition 

We now investigate the Mott transition at half filling. Figure U] shows U- 
dependence of the double occupancy D occ . = (n^n^) for several choices of tempera- 
ture. At high temperatures, D occ . smoothly decreases as U increases, which indicates 
that local spin moments are developed. As the temperature is lowered, a singular 
behavior emerges around characteristic values of U. When 0.014 < T/W < 0.05, 
-D occ . shows a crossover around U/W ~ 1.0-1.4. For reference, in Fig. [5l we show 
dD occ JdU , which is computed by numerical differentiation of D occ .. The maximum 
of dD occ JdU can be identified as the metal-insulator crossover. Therefore, we de- 
fine the crossover point U* by U that gives a maximum of dD occ /dU. At lower 
temperature T/W = 0.0125, the crossover evolves to a discontinuity accompanied 
by hysteresis, which characterizes a first-order phase transition at U c /W ~ 1.37. 
We thus end up with the phase diagram shown in Fig. [3l where the critical end 
point is located at U/W ~ 1.36 and T/W ~ 0.014. We note that U c is much larger 
than the crossover strength of U found for the unfrustrated square lattice model. 79 > 
As is the case for the triangular lattice, 2l > the double occupancy D occ . increases in 
the metallic phase (U < U c ) as T decreases, while it is almost independent of T in 
the insulating phase (U > U c ). The increase of -D cc. at low temperatures implies 
that local moments are suppressed due to the itinerancy of electrons, which in turn 
leads to the development of coherent quasiparticle dynamics. It should be noticed 
that in the metallic phase near the critical point, D cc. starts to increase at very 
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low temperatures. This means that the coherence temperature To characterizing 
quasiparticle formation is very low. This naturally gives rise to strong frustration 
and brings about unusual metallic properties near the Mott transition, as we will see 
momentarily below. 

In order to clearly examine quasiparticle formation around the Mott transition, 
we compute the density of states (DOS), — ^ImGi oc (u> + iO). This is calculated from 
the imaginary time single-electron Green's function G\ oc (r) = — (T r Cj o -(r)c| o .(0)) for 
real frequency to by applying MEM. In Fig. [6j we show DOS at T/W = 0.0125 for 
several choices of the interaction strength U /W . At U = 0, DOS has three distinct 
energy bands including a 5-function peak above the Fermi level. As U /W increases, 
DOS forms heavy quasiparticle peaks around the Fermi level and finally develops 
a dip at U/W ~ 1.40, signaling the Mott transition. We find two characteristic 
properties in the metallic phase close to the transition point. First, the heavy quasi- 
particles survive up to the transition point (U/W = 1.30 and 1.36) and there is no 
evidence for pseudo-gap formation, in accordance with the U- and T-dependence 
of double occupancy in Fig. 01 This is related to the suppression of magnetic in- 
stabilities in our system, in contrast to the square lattice case, where quasiparticle 
dynamics are strongly incoherent and a pseudo gap opens. The second point is a 
large renormalization of quasiparticle weight near the transition point. We can see 
three renormalized peaks near the Fermi level: not only the peak near the Fermi 
surface but also the two other bands away from the Fermi surface are renormalized 
to participate in quasiparticle formation. 

Such evolution of quasiparticles can be also clearly seen in the T-dependence of 
DOS shown in Fig. In the insulating phase (U/W = 1.5), there is a dip struc- 
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Fig. 8. (Color online) Nearest-neighbor spin 
correlation function (SfSf +1 ) as a function 
of U/W. 



Fig. 9. (Color online) Temperature depen- 
dence of (SiSi +1 ) for several values of 
U/W. 



ture near the Fermi level already at quite high temperatures and it becomes more 
prominent with lowering T, and eventually a gap opens at low temperatures. On 
the other hand, in the metallic phase close to the Mott transition (U/W = 1.3), the 
quasiparticle peak develops as T decreases instead of the pseudo-gap formation. The 
three quasiparticle peaks evolve near the Fermi level with lowering T, although there 
exists a dip instead of peak at high temperatures. Therefore, the three quasiparticle 
bands are all relevant for low-energy excitations near the Mott transition, in contrast 
to the weak coupling regime where only the single band around the Fermi surface is 
relevant. 

3.2. Anomalous spin correlations in the metallic phase 

It should be noted that quasiparticles exhibit anomalous spin correlations due 
to strong frustration around the transition point. We show the nearest-neighbor 
spin correlation function (SfSf +1 ) at several temperatures in Fig. [8j Here, Sf = 

(cLcjf — cjiCj|)/2. It is seen that {SfSf +l ) is always negative so that the spin correla- 
tion is antiferromagnetic (AF), which gives rise to strong frustration on the kagome 
lattice. With increasing U/W, the nearest-neighbor AF spin correlation is gradually 
enhanced. In the insulating phase the AF spin correlation becomes stronger as T 
decreases. We note that the low-temperature spin correlation in the insulating phase 
is weaker than that in isolated triangle, {SfSf +1 ) = —1/12. More striking behavior 
appears in the metallic phase near the transition point: the AF spin correlation is 
once enhanced and then suppressed with decrease of temperature, as shown in Fig. 
121 This anomalous temperature dependence comes from the competition between 
the quasiparticle formation and the frustrated spin correlations, which is charac- 
terized by two energy scales: the coherence temperature To and Tm characterizing 
the AF spin fluctuations. The AF correlation enhanced around T ~ Tm stabilizes 
localized moments and gives rise to frustration, which is consistent with the mono- 
tonic enhancement of spin correlations in the insulating phase in Fig. On the 
other hand, when the system is in the metallic phase, electrons recover coherence 
in itinerant motion below To. Frustration is thus relaxed by itinerancy of electrons 
through the suppression of AF correlations at T < To- Therefore, the nonmono- 
tonic temperature-dependence in (S?Sf +1 ) demonstrates the formation of the heavy 
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co/W co/W 

Fig. 10. (Color online) Dynamical susceptibility Imxioc(w) for several values of U/W at T/W = 
0.0125. 



quasiparticles under strong frustration effects. 

We can see the anomalous properties also in dynamical spin correlations. We 
compute the dynamical spin susceptibility defined by 

Xi oc (u)=i dte^([Sf(t),Si(0)]). (3-2) 
J o 

Shown in Fig. [10] is Imxi oc (u>) around the Mott transition at T/W = 0.0125. Note 
that lnrxioc(<-<'0 dramatically changes its profile around the Mott transition. In the 
insulating phase (U/W = 1.4), there are two distinct peaks in Imxioc(^) at low 
energies. On the other hand, in the metallic phase (U/W = 1.3, 1.36), two peaks 
are renormalized into a single peak and its peak value is strongly suppressed. This 
is a clear demonstration of drastic change in spin dynamics between metallic and 
insulating phases in frustrated systems. The double-peak structure in the insulating 
phase is due to the dominant short-range AF correlations at low temperatures. The 
strongly enhanced low-energy peak in xiocfa) corresponds to excitations among the 
almost degenerate states for which a singlet spin pair is formed inside the unit cell, 
while the higher-energy hump is caused by the excitations from these low-energy 
states to other excited states. In the metallic phase, the AF correlations are sup- 
pressed and then frustration is relaxed via the itinerancy of electrons, which leads to 
the renormalized single peak structure in Xioc( w )- We can thus say that the dramatic 
change in Xioc( w ) characterizes the competition between itinerancy and frustration 
of correlated electrons around the Mott transition. 

3.3. Enhanced one- dimensional spin correlations 

We now investigate how these local spin correlations affect long range correla- 
tions by calculating the q-dependence of static susceptibility, 




where 7,(5 = 1,2,3 denote the superlattice indices. We compute x-ys(<i), following 
the procedure explained in the previous section, where nearest-neighbor as well as 
local vertex corrections are included. Let us introduce x m (q) for three normal modes 
(xi(l) > X2(q) > X3(q)) defined by eigenvalues of the 3x3 matrix X'ysfa)- We show 
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Fig. 11. Wave-vector dependence of the static susceptibility Xm(q) f° r several values of U/W at 
T/W = 0.0333. The three-dimensional plots of Xm(q) are shown in the upper panels, from top 
to bottom, m = 1, 2, 3. The two-dimensional plots in the lower panels show the dominant mode 
of the susceptibility xi( c l) m the upper panels. Hexagons in figures denote the first Brillouin 
zone, as shown in Fig. [2jb). 




Fig. 12. (Color online) Enhanced ID spin correlations in the insulating phase: U/W = 1.4 at 
T/W = 0.0333. 

X m (q) for several choices of U/W at T/W = 0.0333 in Fig. [TTJ The maximum 
eigenvalue Xii^) has much weaker q-dependence than that for the other two modes, 
while the second largest mode X2(q) has a strong q-dependence with a maximum 
at q = (0,0). We note that these results are in accordance with those obtained by 
FLEX 13 ) and also by QMC approaches. 14 ) However, we find more striking features 
in the strong coupling regime. At U/W = 0, Xilq) has a maximum at six points 
in the Brillouin zone. With increase of U/W, Xi( c l) is enhanced not only at the six 
points but also on the three lines through T and M points. Thus the q-dependence 
of Xilq) becomes much flatter at U/W = 1.1 than in the {7 = case. Once the 
system enters the insulating phase, the q-dependence of xi(q) dramatically changes 
its character due to the enhancement of short range AF correlations. At U/W = 1.4, 
the susceptibility Xi( { l) further grows along the three lines in q space and becomes 
dominant there instead of the six points that give the leading magnetic mode in 
the weak coupling regime. Furthermore, the analysis of the eigenvectors of Xi( { l) 
concludes that two spins in the unit cell are antiferromagnetically coupled but the 
other spin is free. This implies that the enhanced spin fluctuations favor a spatial 
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Fig. 13. (Color online) Sketch of the anisotropic Fig. 14. (Color online) Phase diagram of Hub- 
triangular lattice bard model on anisotropic triangular lattice 

for t'/t = 0.8. 



spin configuration in which one-dimensional (ID) AF-correlated spin chains are in- 
dependently formed across free spins in three distinct directions. We illustrate a 
schematic picture of the three equivalent types of enhanced spin correlations in Fig. 
[T2l This is one of the naturally expected spin correlations on the kagome lattice, 
since it stabilizes antiferromagnetic configurations in one direction, which is more 
stable than the naively expected spin configuration having a singlet pair and a free 
spin in each cluster. We wish to emphasize that the ID correlations found here in 
the finite-temperature Mott insulating phase are different from those for the Heisen- 
berg model on the Kagome lattice with the nearest-neighbor exchange obtained by 
both classical and semi-classical approximations, 81 )' 82 ) but are similar to the q = 
structure predicted for the classical Heisenberg model with a further neighbor ex- 
change. 81 ) The essential difference is that there is almost no correlation between 
the different chains in our results for the Hubbard model. These ID correlations 
have been recently studied by Udagawa and Motome by means of the larger-cluster 
CDMFT. 84 ) They have clarified the origin of the ID correlations in terms of the 
strong renormalization effects of electrons at finite temperatures. 

Summarizing this section, we have clarified that the metallic phase is stabilized 
up to fairly large U in electrons on the strongly frustrated kagome lattice, resulting 
in the three-band heavy quasiparticles. We have seen that this gives rise to several 
anomalous properties of spin correlation functions in the metallic phase close to the 
Mott transition point. As a characteristic of the kagome lattice system, novel ID 
spin correlations appear in the insulating phase at intermediate temperatures. With 
approaching zero temperature, the spin liquid state or other nonmagnetic ordered 
states are expected to be realized at low temperatures. 10 )- 83 ) 

§4. Anisotropic triangular lattice system 

In this section, we study the finite-temperature Mott transition in the Hubbard 
model on the anisotropic triangular lattice. We determine the U-T phase diagram 
as shown in Fig. [l"4"l which shows a remarkable property, i.e. reentrant behavior in 
the Mott transition. 

Let us consider the Hubbard model (|2Tp on the anisotropic triangular lattice 
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shown in Fig. [T3J, 








t (t > 0) (site i an j : nearest neighbors) 
t 1 (t' > 0) (site i an j : diagonal) 
(otherwise) 



(4-1) 



We use the band width W as an energy unit for given anisotropy t'/t, and it is 
W = 4(t + i') + t 2 /i' f or f > 0.5t and W = 8t for f < 0.5*. To analyze this 
model, we use the four-site cluster CDMFT. Considering four sublattices labeled by 
1-4, as shown in Fig. [T3l we map the original lattice model to a four-site cluster 
model coupled to the self-consistently determined medium. Fifty-times iterations 
m a CDMFT procedure and typically 10 6 QMC sweeps with Trotter time slices 
L = \2t(3 are performed to obtain numerical convergence. 

4.1. Mott transition 

We investigate the Mott transition for the model (|2-ip at half filling. To this end, 
let us first compute the T-dependence of the double occupancy D occ . = (n^n^) for 
typical interaction strength U/W, where the ground state is insulating and also in 
the vicinity of Mott transition point. We find a remarkable property in our frustrated 
system, i. e. nonmonotonic T-dependence of D occ . . It is seen in Fig. [15] that -D cc. 
decreases at high temperatures, and then exhibits an upturn in the intermediate 
temperature region having a local minimum at T/W ~ 0.06, as T decreases. At much 
lower temperatures, D occ , starts to decrease again and thus shows a hump structure. 
The nonmonotonic T-dependence of D occ _ implies that our system once changes 
from insulating to metallic and then reenters the insulating phase as T decreases. 
This is quite different from that known for the infinite dimensional Hubbard model, 
where T> occ . (T) has a single minimum at the Fermi- liquid coherence temperature To. 
In the latter model, the system is insulating at T > To and Fermi liquid like at 
T < To. 29 - 1 The nonmonotonic behavior found here is also different from that for the 
unfrustrated square lattice Hubbard model. In the square-lattice Hubbard model, 
the Fermi-liquid coherence is disturbed by the antiferromagnetic (AF) interaction 
due to the perfect nesting of Fermi surface, which results in monotonic decrease of 
-Docc. - 68 ^' 79 ^ It is seen that the hump structure in -D cc. becomes more prominent and 
shifts to lower temperatures as t! increases, although it is less visible for t'/t = 0.5. 

We can see such characteristic behavior also in DOS as shown in Fig. [T6j This 
is wave- vector-integrated electron spectral function. For t'/t = 0.5, DOS shows 
a dip structure near the Fermi level at high temperatures. As T decreases, the 
dip becomes prominent, and finally a gap opens clearly in the low-T insulating 
phase. On the other hand, for t'/t = 0.8 with stronger frustration, the nonmonotonic 
behavior appears in DOS. The quasiparticle peak develops near the Fermi level with 
lowering temperature, although the dip structure appears at high temperatures. 
As T further decreases, the quasiparticle peak disappears and an insulating gap 
opens again. These properties are consistent with the results of D occ . Therefore, it 
is concluded that the nonmonotonic behavior is a characteristic feature caused by 
geometrical frustration. 

In order to clarify whether the change between metal and insulator is a real phase 
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Fig. 15. (Color online) Nonmonotonic temperature dependence of double occupancy in the typical 
strong-(7 regime U ~ W, where ground states are insulating and in the vicinity of the Mott 
transition. 
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Fig. 16. (Color online) Temperature dependence of DOS for t'/t = 0.5 (a) and t' /t = 0.8 (b). For 
t'/t = 0.8, the metallic quasiparticle peak appears near the Fermi level at the intermediate 
temperature T/W = 0.04734. 



transition or crossover, we examine the double occupancy for typical anisotropy 
t'/t = 0.8 with varying U. Let us start from the noninteracting system to reach 
the large-?/ regime, typically U/W ~ 1.2, and then calculate 73 occ . with gradually 
decreasing U. It is seen in Fig. [T7] that the double occupancy jumps at critical 
interaction strength U c as U decreases, signaling a first-order Mott transition. The 
size of the jump shrinks as T increases, and eventually vanishes above T/W ~ 0.035. 
It is expected that the critical end point is located at T/W ~ 0.035 and U/W ~ 
1.077. At high temperatures, the system exhibits a crossover between metal and 
insulator, where we define the boundary by the temperature at which the double 
occupancy takes the first local minimum at high temperatures, as seen in Fig. [151 
Note that the metal-insulator boundary is in accordance with that determined by 
the local minimum of the density of states at the Fermi energy. We thus end up with 
the phase diagram shown in Fig. Q31 We wish to emphasize here that U C (T) in our 
system has a slope with the opposite sign to the behavior in the infinite dimensional 
model at low temperatures, whereas the high-T crossover exhibits similar behavior 
to the infinite-dimensional case. 
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Fig. 17. (Color online) Double occupancy as a function of interaction strength U/t for t' /t = 0.8 
at several temperatures. We show only the transition from insulator to metal with weakening 
U (defined as U c i), although we find hysteresis. 

4.2. Reentrant behavior 

It is known that the single-site DMFT gives entropy of about In 2 per site in 
the paramagnetic insulating phase corresponding to localized free spins and smaller 
entropy is realized in the metallic phase with the Fermi-liquid coherence. Therefore, 
near the Mott transition temperature, the system is insulating at high temperatures 
to gain large entropy and the metallic phase exists always in the lower temperature 
region. 29 ^ It should be noted, however, that the spatial fluctuations, which are not 
taken into account in the single-site DMFT, are important at low temperatures. 
For instance, according to the dynamical cluster study of the square-lattice Hub- 
bard model, the Fermi-liquid metallic phase does not appear because of strong AF 
correlations. 68 )' 79 ) In contrast, the magnetic correlations in our system are hard to 
develop until low temperature T/W ~ 0.05 due to strong geometrical frustration. 
Hence, when the temperature is lowered, the entropy is released not by spin corre- 
lations but by the itinerancy of electrons at T/W > 0.05, which gives rise to the 
crossover behavior from insulator to metal as shown in Fig. [TD The emergence 
of such Fermi-liquid states is one of the characteristics in the vicinity of the Mott 
transition with geometrical frustration. 73 ) Note that the magnetic correlations are 
enhanced at much lower temperatures, finally triggering a first-order phase transition 
from the Fermi liquid to an insulator with smaller entropy. Hence with decreasing 
T, U C (T) decreases at low temperatures (T/W < 0.05) in contrast to the behavior 
at higher temperatures (T/W > 0.05). We can thus say that the reentrant Mott 
transition found here for the anisotropic triangular lattice is caused by the competi- 
tion induced by geometrical frustration between the Fermi-liquid formation and the 
magnetic correlations. 

Let us now discuss the momentum resolved single-electron spectrum A^uj), 
in which we can clearly see the development of the quasiparticles and magnetic 
correlations discussed above. This is calculated by using Eq. (|2-7p and MEM. We 
show A k (u) for U/W = 0.9468, t'/t = 0.8 at typical temperatures in Fig. [TgJ It 
is seen that (uj) shows an insulating behavior at high temperatures, where it has 
a large Hubbard gap of order of U /W and no quasiparticle peak. With decreasing 
temperature, a quasiparticle peak starts to develop inside the gap, and results in 
a quasiparticle band with weak dispersion. This clearly suggests the appearance of 
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Fig. 18. (Color online) Momentum resolved single-electron spectrum A^(ui) for U/W = 0.9468, 
t' /t = 0.8 at several temperatures. 



the frustration-induced metallic phase. The emergence of the metallic phase due to 
geometrical frustration is in accordance with the previous studies of the Hubbard 
model on the triangular lattice 18 )' 68 ) and the kagome lattice. 73 ) As temperature 
further decreases, the quasiparticle peaks split and acquire a very small gap, and the 
system enters another insulating phase again. Note that the small gap is caused by 
the exchange interaction among quasiparticles, which is consistent with the results 
of the CDMFT study with exact diagonalization method at zero temperature. 24 ) It 
is thus confirmed that the present frustrated system exhibits the insulator-metal- 
insulator reentrant behavior with decreasing temperature. 

4.3. Magnetic instability 

Here, we wish to briefly discuss magnetic instability. The static spin suscep- 
tibility x(q) computed for typical U /W and T/W around the Mott transition is 
shown in Fig. [191 For t' jt < 0.7 (weak frustration), it is seen that xfa) develops 
a peak structure with lowering T at q = (tt, it) corresponding to the commensurate 
antiferromagnetism. The behavior is consistent with recent studies of the Hubbard 
model around the Mott transition. 17 )' 22 )' 27 ) Quite different tendency appears at 
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t'/t = 1.0 (triangular lattice), where the spin susceptibility x(q) has a maximum at 
q = (27r/3, 27r/3) corresponding to the 120° structure, and the development of x(q) 
is much slower than the AF correlations for small t'. For U/W = 1.065, t'/t = 0.8 
at T/W = 0.02367, where the system is in the insulating phase near the Mott tran- 
sition point, x(q) does not diverge, but takes a maximum at incommensurate wave 
vectors q ~ (0.77T, 0.7?r). It is remarkable that x(q) remains finite just below the 
first order Mott transition temperature and hence the paramagnetic Mott insula- 
tor is not precluded by the magnetically ordered phase. This is contrasted to the 
single-site DMFT results: the magnetic order conceals the Mott transition. 29 )> 38 ) 
On the other hand, the magnetic ordering in our case, which is strongly suppressed 
by frustration effects, emerges below the Mott transition temperature, although a 
finite-temperature magnetic transition is due to a mean-field type approximation. In 
this way, the magnetic correlations play a crucial role in driving the Mott transition 
at low temperatures, although they do not trigger a real magnetic instability. 

4.4. Controlling frustration 

It is to be noted that the determined phase diagram (Fig. [T4"|) in the U-T 
plane for t'/t = 0.8 is qualitatively consistent with the experimental data 16 ' in k- 
(BEDT-TTF) 2 Cu[N(CN) 2 ]C1 with t'/t ~ 0.75. Now, a question naturally arises: 
what happens if we control frustration by changing the ratio of t'/t. To this end, 
we have systematically studied the Mott transition for different choices of t'/t. For 
weakly frustrated case t'/t < 0.5, we have found that the Fermi-liquid formation is 
suppressed by the strong AF correlations, and thus the reentrant Mott transition 
becomes less prominent. In this case, the Mott transition temperature is very low 85 ) 
and the AF long range order hides the Mott transition within CDMFT approach. 
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For fully frustrated case t' ~ t, on the other hand, the Fermi-liquid states are well 
stabilized by frustration and the metallic region is extended, because the magnetic 
fluctuations of the 120° structure are weak. We thus expect that the low-temperature 
Mott transition line shifts to lower-temperature regime. 21 ) Note that the above 
tendency is consistent with the experiments on another organic material k-(BEDT- 
TTF)2Cu2(CN)3 with t' ~ where any magnetic order was not observed so far 
experimentally. We conclude that the reentrant behavior can be observed most 
clearly in electron systems with moderate frustration. 

Summarizing this section, we have found novel reentrant behavior in the Mott 
transition in the anisotropic triangular lattice Hubbard model, and have clarified 
that the reentrant behavior in the Mott transition is caused by the competition 
between the Fermi-liquid formation and the magnetic correlations under geometrical 
frustration. 

§5. Summary 

We have studied the Hubbard model on the kagome and anisotropic triangular 
lattice by means of the cellular dynamical mean field theory. Through the system- 
atic studies, we have elucidated some characteristic properties common to frustrated 
electron systems near the Mott transition. First, we have shown that the metallic 
phase is stabilized up to fairly large Hubbard interactions under strong frustration 
in both models. This naturally leads to the formation of strongly renormalized 
heavy fermions near the Mott transition point, where some anomalous properties 
are caused by almost localized electrons with strong geometrical frustration. One of 
such anomalous features emerges in the nonmonotonic temperature dependence of 
the spin correlation function found for the kagome lattice model around the Mott 
transition. This is indeed due to electron correlations strongly influenced by frus- 
tration. We have also found that such anomalous spin correlations result in more 
striking behavior in the Mott transition in the case of the anisotropic triangular lat- 
tice system. Namely, the Mott transition shows novel reentrant behavior due to the 
competition between Fermi-liquid formation and magnetic correlations, as typically 
seen for the triangular lattice system for if /t = 0.8. 

Although we have used the small-cluster CDMFT in this paper, we believe that 
the anomalous properties found here for the Mott transition are robust, which may 
not qualitatively change even for a larger cluster size, since the competition be- 
tween the Fermi-liquid formation and magnetic correlations should occur generally 
in frustrated electron systems, Therefore, we expect that nonmonotonic tempera- 
ture dependence of the spin correlation function and/or the reentrant behavior in 
the Mott transition will be observed experimentally in a variety of frustrated electron 
systems. 
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